Add breeze ci audit for the fork CI setup - #74823

…74819, #74820, #74822, #74823, #74825, #75578, #75588, #75611) (#75618)

* Fail the local yml mounts prek hook when synchronization fails (#73785)

* Fail the local yml mounts prek hook when synchronization fails

The hook printed a red error and exited zero, so a failed synchronization
passed as green and left the generated mounts in local.yml stale. The
failure it is most likely to hide is a broken docker_command_utils.py,
which is one of the two files that trigger the hook in the first place.

* Run the real prek helper in the local yml mounts test

Stubbing common_prek_utils made the test carry a second copy of the
script's imports and a copy of the script itself, and left the helper
the hook actually runs untested. A fake breeze on PATH is enough.

(cherry picked from commit eddc34c)
Generated-by: Claude Opus 5

* Reduce K8s test resource usage with single-node Kind clusters (#73998)

* Reduce K8s test resource usage with single-node Kind clusters

* Add test for forwarded node port matching Kind config and NodePort service

* Refactor Kubernetes configuration and tests to use centralized port rendering

(cherry picked from commit a5e5e1a)
Generated-by: Claude Opus 5

* Stop parallel prek hooks racing on __pycache__ cleanup (#73823)

* Stop parallel prek hooks racing on __pycache__ cleanup

prek runs hooks in parallel and every breeze invocation removes all
__pycache__ directories in the checkout on start-up. Any other hook
process importing from the source tree (breeze itself, prek hook
scripts, local mypy runs) can write a fresh .pyc between rmtree emptying
a directory and removing it, which fails the breeze hook with
"OSError: [Errno 39] Directory not empty" - a flaky Static checks
failure in CI.

Redirecting bytecode of every hook into a hidden, gitignored .build/
directory keeps the source tree free of new .pyc files while hooks run,
and a new check keeps future hooks from reintroducing the race. The
cleanup additionally tolerates ENOTEMPTY for writers prek does not
control, such as an IDE.

Generated-by: Claude Opus 5

* Set PYTHONPYCACHEPREFIX on the hooks added since the branch was cut

Eight hooks that landed on main after this branch was created did not redirect their bytecode, so the new check-prek-hooks-pycache-prefix hook failed after the rebase.

Generated-by: Claude Opus 5

* Update scripts/ci/prek/check_prek_hooks_pycache_prefix.py

Co-authored-by: Wei Lee <hello@wei-lee.me>

---------

Co-authored-by: Wei Lee <hello@wei-lee.me>

(cherry picked from commit 0cde487)

* Add the fork CI gate to ci-amd.yml (#74819)

* Add design and ADR for running CI in contributor forks (AIP-120)

AIP-120 moves CI for external contributions into contributor forks so
project runners serve committers, collaborators and exempted
organisations, and so maintainers only review pull requests that
already passed in the author's fork. The mechanics span the main test
workflow, a scheduled reconciler for gated pull requests, a commit
status that surfaces the fork CI state with a link to the fork run, a
local pre-flight command and the contribution docs, so the design is
recorded before implementation and the decisions that shape it are
captured as ADR 0018.

Generated-by: Claude Opus 5

* Close gated pull requests that tamper with the fork CI gate

The gate job in ci-amd.yml runs from the pull request's merge commit,
so a gated external pull request can drop the gate or add a workflow
with a pull_request trigger and get project CI anyway. The design
accepted that as a one-run cost but left it unbounded per pull
request, because every further push would get another run.

The reconciler now detects this after the fact from what GitHub
recorded rather than from the pull request tree: project CI ran for
the head SHA and the pull request changes files under
.github/workflows/. Both signals are required so that legitimate
project CI runs (backlog reset, a removed "use project ci") and
legitimate workflow edits that keep the gate intact are never closed.
Closing stops further pull_request runs, so one attempt buys one run,
and the closure is reported to Slack as an attention item rather than
as a reconciler failure. ADR 0018 records the trade-off, including the
external pull requests with wanted workflow changes that must keep
"use project ci".

Generated-by: Claude Opus 5

* Let volunteer contributors opt in to a fork CI pilot

The dev list agreed to start with an opt-in pilot by a few volunteer
contributors rather than enabling fork CI for everyone at once. The
committer-only "use fork ci" label cannot serve that: it arrives after
gpgsig -----BEGIN PGP SIGNATURE-----

 wsFcBAABCAAQBQJq0YVxCRC1aQ7uu5UhlAAAcYgQACYDDrIdXBZxVhqczF3TNrO7
 qdkPgQy3NgLfVgmY0iy7ypzDRYY+glMD415Qc5OJy7n5M7XBqJ3ALQf04U/IGm0O
 bMXgq1+Z9B/sRz27VNQYJm9X30hL+pE4iYHwGAQHXsr/TIgl2WN+V4awsRYEVQXM
 x3Q/jwR12/YbiW2R3c61J+c7mrLdw4bZ6UtnDQqWUYkimgRGDLFvlZ/yMAj8kBuU
 UyXGeOtTaIJHlghczgxlTeBDNMjuE9o5S6xsW5n+eWJXy7E/883L6uMWR3OjLR01
 l2gpSqrpi52xYMVd8Zik+gpuWjkQ45C9k+XgGQfLfhoelZJ8uZfToMqARne4xVlZ
 4hFIHfZSGdcLCi+sbU6PKxrMdX7HF4vDAxP9Tv4IpIzANRxG9IH5uXpHwAAx24LF
 ogZQDyykjqTsadl7SqjKDw/Hybv4hklnnr8BFfvYd54iFkkP3fqSizlV4ecr1tyJ
 OBvI7DbNNNdySebmnMYQGBziLQF5uUseD8srhCiXG5XNaSqbOnNXc+FHZmn0oU3O
 EJ15O0QPi4rkRXQ5gYCjjb4tO99H3Mw6bMgiIWTjVa/1QPmOb9mtEbpRP/ESHuOe
 1VgYd8tUczg6FB6Po5f/DXV2rZB36ij9Bo4nBmR7V1HApOx4E88lDIznGpqP/cWf
 MetUk2Nom7/sr+UaBu3i
 =wncY
 -----END PGP SIGNATURE-----

the first pull_request run has already used project CI, and a
maintainer would have to label every pull request the volunteer opens.
A per-login join instant gates all their new pull requests from the
first run while leaving older ones on project CI, so those can never be
mistaken for gate tampering.

The reconciler also has to run while the gate is disabled, which the
earlier "exits early" wording contradicted for the label pilot as well.

Generated-by: Claude Opus 5

* Close gaps in the fork CI design found in review

Enabling ci-amd.yml in a fork also enables its cron, which would run the
full canary matrix in every contributor fork twice a day. A join instant
or enabled_since earlier than the merge that sets it gates pull requests
that already ran project CI. Forks not named "airflow" would never show
a fork run, and a pull request reopened after an inactivity close would
be closed again at once. ADR 0018 was already taken.

Generated-by: Claude Opus 5

* Trust fork CI only for pull requests that cannot change it

A fork run executes the workflows, breeze and CI scripts of the pushed commit, so a pull request changing them decides its own fork CI result. Gate tampering was only checked on the head commit, so pushing a tampered commit and then a clean one hid the bypass, and the check could be dodged by renaming a job to the gate's name or by truncated check-suite listings. The gate also fell back to project CI when its label read failed, which an exhausted token quota could trigger for every pull request.

Generated-by: Claude Opus 5

* Add fork CI eligibility function and configuration

The gate job and the reconciler must agree on which pull requests use project CI and which are left to fork CI, so the decision lives in one standard-library-only function that the gate can run without installing breeze, driven by one reviewed configuration file. It ships disabled.

Generated-by: Claude Opus 5

* Gate project CI for external pull requests in ci-amd.yml

Contributors must get CI from a plain push to their fork, and project
CI must not run for pull requests left to fork CI, without reacting to
pull request events with write permissions. A gate job ahead of
build-info decides from the base branch's configuration, and the push
trigger widens to every branch while the gate keeps the old branch list
in effect for apache/airflow.

Skipping build-info also skips everything that depends on it, which the
always() conditions of finalize-tests and notify-slack would otherwise
treat as success. Pilot join instants are compared with pull request
creation but only take effect on merge, so the config hook keeps new
ones in the future.

Generated-by: Claude Opus 5

* Address review of the fork CI gate

Drop the collaborator-permission lookup: it needs push access, which neither the gate's nor the monitor's token has, so it never matched. A committer with private apache membership makes it public or is allowlisted.

Take the gate from the default branch for pushes, so a push to a branch without the gate files keeps skipping project CI instead of looking like base_without_gate. Check the PR title in the gate job for gated pull requests, which skip ci-image-checks. Check every pilot entry when the base config does not parse, and restrict the E2E flaky-tests report to scheduled runs now that every push to main creates a gate-only run.

Generated-by: Claude Opus 5

* Anchor the canonical push branches on the pull request filter

Since #75420 ci-arm.yml has no push trigger: release branches get their full runs from release-branch-ci.yml. The canonical push branch filters are now checked against the pull_request filter of ci-amd.yml without main and the -stable branches, which is the set release-branch-ci.yml uses too.

Generated-by: Claude Opus 5

* Set PYTHONPYCACHEPREFIX for the fork CI gate config hook

main now requires every prek hook to write bytecode outside the source tree (check-prek-hooks-pycache-prefix).

Generated-by: Claude Opus 5

* Add the fork CI gate to the needs of CI passed

Generated-by: Claude Opus 5

#75142 added the CI passed job that auto-merge waits for, and check-ci-gate-needs requires every CI job in its needs. The gate exists only in ci-amd.yml, so check-ci-workflows-in-sync allows that one line as an AMD-only divergence.

* Describe the fork CI gate in the CI passed docs as in place

The gate now exists, so the CI passed comments and the auto-merge docs no longer say what happens until it lands.

Generated-by: Claude Opus 5

* Skip the scheduled ARM canary in forks

A fork that enables Actions for fork CI also gets the ci-arm.yml cron, and
nothing stopped its canary from running there. The check-run-needed step
shared by both workflows now reports run-needed=false outside apache/airflow.
Also correct the canonical push branch filter source in the design doc and
reuse AIRFLOW_ROOT_PATH in the gate config hook.

Generated-by: Claude Opus 5

(cherry picked from commit 3ce8be4)

The README conflict was resolved by keeping the release branch's newsfragment-check sentence.

* Run fork pushes with pull-request selective checks (#74820)

* Run fork pushes with pull-request selective checks

A push in a contributor fork would otherwise be treated as a post-merge push and run the full matrix, about 220 minutes on the contributor's 20-runner cap, for every push. Computing the change set against apache/airflow gives the same selection a pull request would get, independent of how stale the fork's main is, and fork runs must never select self-hosted runners that forks do not have.

Generated-by: Claude Opus 5

* Read pull request labels and the merge base for fork pushes

A fork push carries no labels, so full tests needed, allow provider dependency bump and the other labels did nothing in fork CI and a PR that needed one stayed red. Read them from the open apache/airflow pull request of the pushed branch.

The pyproject.toml and provider dependency bump checks compared the pushed commit with its parent, which after a merge of upstream main is the contributor's own previous tip, so a release manager's bump from main counted as theirs. Read the previous version at the compare's merge base instead.

Generated-by: Claude Opus 5

* Match the fork pull request by base when reading its labels

One head branch can have open pull requests into several bases, so the
label lookup now filters on the branch line the changes are compared against.
The fork push selective-check tests move to test_ci_commands.py, next to the
other selective_check command tests.

Generated-by: Claude Opus 5

(cherry picked from commit 164a812)

* Add the fork CI reconciler and its scheduled monitor (#74822)

* Add the fork CI reconciler and its scheduled monitor

Gated pull requests need labels, draft state, a state comment, a commit
status that links to the fork run, and closing after inactivity or gate
tampering, and none of that may be written in reaction to an untrusted
pull_request event. A scheduled, idempotent reconciler that derives every
change from what GitHub recorded is the only writer, so missed or
repeated runs are harmless and nothing runs pull request code with write
permissions. Whatever it cannot classify is reported to maintainers
instead of being guessed.

The monitor needs the breeze setup action, which frees disk space with
sudo and docker, so it runs on ubuntu-latest rather than ubuntu-slim.

Generated-by: Claude Opus 5

* Report CI passed from fork CI and address reconciler review

The merge ruleset requires a CI passed check, which gated pull requests never get from project CI. The reconciler now posts a CI passed commit status when the head commit's fork CI is green and trusted, and overwrites it with pending when the commit leaves green, for example after a failed re-run in the fork.

Also from review: drop the collaborator-permission lookup the monitor's token cannot make; stop posting green_unverified to Slack, since the needs project ci label lists those pull requests; judge only completed runs and finished jobs for gate tampering, so a job waiting for a runner does not look like project CI work; read compare files from the first page only; let gh's partial GraphQL output through when it exits non-zero; and warn in the state comment when a green branch is far behind its base.

Generated-by: Claude Opus 5

* Do not count jobs every gated run starts as gate tampering

Since #75420 every ci-amd.yml run starts check-run-needed, and once the CI passed gate of the auto-merge ruleset is in, a gated run also starts it as "CI passed (project CI not run)". The tamper detector counted any started job other than the gate as project CI work, so it would have closed every gated pull request. Those two jobs are now expected, under the same 5-minute limit as the gate.

Generated-by: Claude Opus 5

* Count only jobs a runner picked up as fork CI gate tampering

A run cancelled by a newer push before its gate started marks every job
completed, so the tampering check read the never-started jobs as project CI
work and could close a gated pull request that edits workflows legitimately.
A job now counts only when it has a runner or steps, and "CI passed", which
such a run reports under its plain name, is bounded like the gate.

Statuses are read from the head commit when the pull request's last listed
commit is another one, and the docs no longer claim a backlog reset never
closes a pull request that changes workflows.

Generated-by: Claude Opus 5

(cherry picked from commit bad9f9c)

* Add breeze ci audit for the fork CI setup (#74823)

* Add breeze ci audit for the fork CI setup

Fork CI only works when the contributor's remotes, fork and fork workflow are set up the way the gate and the reconciler expect, and committers must keep fork CI off so their pushes do not run twice. A read-only audit lets contributors verify all of that before pushing and offers the exact fix for anything missing. Remote parsing moves out of ci upgrade so both commands agree on what a correct setup is, and it now matches apache/airflow exactly instead of any URL containing it.

Generated-by: Claude Opus 5

* Address review of breeze ci audit

Decide the role before the remote and fork checks, so a committer working from an apache/airflow clone is not told to set up a fork their pull requests do not use; those checks, and an enabled fork workflow, are informational for roles that run project CI, since a committer trying use fork ci needs the workflow enabled. Report a failed API call as could-not-check instead of a wrong fork or a missing workflow, and recognise ssh host aliases from ~/.ssh/config as GitHub remotes again, as breeze ci upgrade did before.

Generated-by: Claude Opus 5

* Classify only public apache members as committers in breeze ci audit

The gate decides from the pull request's author association, which reports a private member as CONTRIBUTOR, so their pull requests are gated. The audit asked for any membership, accepted disabled fork CI for them and recommended disabling it. It now checks public membership, and tells a private member to make it public or ask for the allowlist.

Generated-by: Claude Opus 5

(cherry picked from commit ce04458)

* Document fork CI for contributors and maintainers (#74825)

* Document fork CI for contributors and maintainers

Contributors need to know how to set up CI in their fork, how to check that setup, and how to join or leave the pilot, and maintainers need to know which parts of a gated pull request's lifecycle the reconciler owns so triage does not fight it. The previous one-line advice to enable GitHub Actions in the fork did not say any of that.

Generated-by: Claude Opus 5

* Say when fork CI applies in the contributor docs

While the gate is off, a contributor outside the pilot who enabled fork CI as the docs said got CI twice on every pull request. The docs now say fork CI applies to pilot participants and use fork ci pull requests until it is switched on for everyone, and the maintainer docs cover the needs project ci label, the CI passed status and why removing fork ci required is not enough.

Generated-by: Claude Opus 5

* Describe apache/airflow push runs after release-branch CI moved to a schedule

ci-arm.yml no longer lists the push branches (#75420). Point to CANONICAL_PUSH_BRANCH_FILTERS and say that the incremental push base is only used in apache/airflow.

Generated-by: Claude Opus 5

* Fix the fork CI section title level and label wording in the contributor docs

The new subsection used a title style the file had not established, which
made Newsfragments a level-4 section and broke links to it. Also say when a
pull request the author drafted gets ready for maintainer review, use a
placeholder pilot instant, and describe the fork compare base as breeze's
default branch rather than the branch name.

Generated-by: Claude Opus 5

(cherry picked from commit bfc2882)

* Do not fail CI passed for a cancelled duplicate CI run (#75578)

* Do not fail CI passed for a cancelled duplicate CI run

A run that concurrency cancels still runs the always() CI passed job, and
that job failed the required check. When a stacked pull request's base and
head are force-pushed together, GitHub sends two synchronize events for the
same commit, so one run is cancelled seconds in and a failed CI passed sits
next to the passing one of its twin. A cancelled run now reports under a
name the rulesets do not require, so nothing is red, while a run that was
only cancelled still never satisfies CI passed and the merge stays blocked.

Generated-by: Claude Opus 5

* Accept the cancelled-run CI passed name in the fork CI tampering check

A cancelled ci-amd.yml run now reports its CI passed job as "CI passed (run
cancelled)". Without the name in the gated-run jobs, a run of a gated pull
request that a newer push cancelled would be taken for project CI work and
close the pull request as tampering with the gate.

Generated-by: Claude Opus 5

* Keep failing CI passed when a CI job times out

A job that hits its timeout is cancelled while the run is not, so the
cancelled-run rename alone would turn a hung job into a passing check
and drop it from the scheduled canary's Slack report.

Generated-by: Claude Opus 5

* Make the Slack canary report wait for the CI passed gate job

Generated-by: Claude Opus 5

(cherry picked from commit 1b9a32b)

The README conflict was resolved by keeping the release branch's "one check" wording.

* Delete the project CI runs that only gate a pull request to fork CI (#75588)

* Delete the project CI runs that only gate a pull request to fork CI

A gated Tests (AMD) run lists every project CI job as a skipped check on
the pull request, next to the fork's result. GitHub cannot delete or
update single Actions check runs, so the fork CI monitor deletes the
whole run once only the gate ran in it.

Generated-by: Claude Opus 5

* Keep gated runs whose commit changes workflows as tampering evidence

Generated-by: Claude Opus 5

* Tell maintainers to close and reopen a gated pull request to start project CI

The fork CI monitor now deletes gated runs, so there is no run left to
re-run after adding use project ci.

Generated-by: Claude Opus 5

* Keep re-run gated runs, whose earlier attempts may have run project CI

Generated-by: Claude Opus 5

(cherry picked from commit daa2a86)

* Let the fork CI monitor gate pull requests and stop repeating Slack alerts (#75611)

GitHub authorises convertPullRequestToDraft and markPullRequestReadyForReview
against the contents permission, and the monitor's token only had read
access. Every attempt to draft a gated pull request failed with "Resource
not accessible by integration", and because drafting was the first mutation,
the state comment, the Fork CI and CI passed statuses were never applied
either: the gate did nothing for the only gated pull request so far.

contents: write also allows pushing to the repository, so it must not reach
the job that installs and runs Breeze and its dependencies. The draft state
is advisory, the required statuses gate merging, so it can be changed by a
separate job that runs nothing but gh and jq.

The same failure was posted to Slack every 15 minutes. A persisting problem
should be posted once, and again only if it goes away and comes back.

Generated-by: Claude Opus 5

(cherry picked from commit 06771a9)